Solid-state imaging devices such as charge coupled device (CCD) image sensors and complementary metal oxide semiconductor (CMOS) image sensors used, for example, in a video camera and a digital still camera realize photo-electric conversion that includes accumulating electric charge in proportion to the amount of incident light and outputting an electric signal proportional to the accumulated electric charge. However, a photo-electric conversion device has its upper limit to its amount of electric charge accumulated, and when a fixed amount of light or more is received, the amount of accumulated electric charge reaches a saturation level. As a result, so-called blown-out highlight occurs that causes subject regions having a fixed level of brightness or more to be set to a saturated luminance level.
One technique of avoiding such a blown-out highlight is a process of controlling sensitivity to its optimal level by controlling a charge accumulation period of the photo-electric conversion device and adjusting exposure time, for example, in accordance with a change in external light. For a bright subject, for example, the exposure time is reduced by releasing the shutter at high speed, reducing the electric charge accumulation period of the photo-electric conversion device and outputting an electric signal before the amount of accumulated electric charge reaches the saturation level. Such a process makes it possible to output an image that accurately reproduces shades of gray appropriate to the subject.
However, releasing the shutter at high speed during shooting of a subject having mixed bright and dark areas gives rise to degraded image quality due to signal to noise ratio (S/N) deterioration resulting from insufficient exposure time in dark areas. In order to accurately reproduce luminance levels of bright and dark areas in a shot image of a subject having mixed bright and dark areas, a process is required to achieve high S/N by using a long exposure time in pixels with only slight incident light on an image sensor and avoid saturation in pixels with much incident light.
Consecutively shooting a plurality of images with different exposure times and merging these images are known as a technique of realizing such a process. That is, the technique generates a single image by using a merging process that includes consecutively and individually shooting long-time exposure images and short-time exposure images and using long-time exposure images for dark image areas and short-time exposure images for bright image areas that may lead to blown-out highlights in long-time exposure images. Thus, it is possible to acquire an image having a wide dynamic range with no blown-out highlights, i.e., a wide dynamic range image (high dynamic range (HDR) image) by merging a plurality of different exposed images.
For example, PTL 1 (JP 2000-50151A) discloses a configuration for acquiring a wide dynamic range image by shooting two images with a plurality of different exposure time settings and merging these images.
However, such a process of using a plurality of images, long- and short-time exposure images, can be used as a process for still images. However, it is difficult to use the process for videos. The reason for this is that shooting long- and short-time exposure images alternately for each of frames making up a video and processing the images result in a variety of problems including lower display frame rate, increased processing burden, and difficulty in achieving real-time display.
If it is necessary to proceed with the process after checking a moving image shot with a camera in real time, for example, as when a surgery is conducted using an endoscope, identicality between shot and observed images, i.e., real-timeness, is required.
Displaying an image after a process of merging a plurality of images as described above leads to delay in displaying the image. Therefore, when real-time display is required as with endoscope, it is not possible to perform the multi-image merging process.